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ABSTRACT 



An optical ranging apparatus is provided which re- 
solves the depth of objects in an object field of a main 
lens. Light is directed by the main lens to a lenticular 
array consisting of an array oflenticules each of which 
generates an image of the lens surface. Each of the 
generated images is directed to a photodetectpr array 
divided into macropixels each of which receives one of 
the lenticule images. Each macropixel is made up of a 
number of subpixels, each of which is a discrete photo- 
detector element and generates an electrical signal pro- 
portional to the intensity of light upon it. A data proces- 
sor receives the signals from the subpixels and assembles 
subimages, each of which consists of one subpixel from 
each macropixel. Each subimage represents a view of 
the entire object field through one portion of the main 
lens. The data processor determines depth of objects in 
the object field by comparing the parallax between 
subimages. A diffuser may be provided between the 
object field and the lenticular array to low pass filter 
light from the object field and reduce aliasing. A field 
lens may also be provided between the main lens and 
the lenticular array to redirect Hght from the main lens 
and prevent skewing of the light reaching the macropix- 
cls. The images from the lenticular array may be re- 
layed to a photodetector array set back from the lentic- 
ular array by using a relay lens. 

44 Oaims, 6 Drawing Sheets 
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FIG. 3A 




FIG. 3B 
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DIGITIZE IMAGE. 



EXTRACT SUBIMAGES CORRESPONDING TO 
DIFFERENT VIEWPOINTS. 



I 



ENHANCE SUBIMAGES BY 
HIGH-PASS FILTERING OR AGO. 



I 



COMPUTE DISPLACEMENT ESTIMATES 
WITH CONFIDENCE IMAGES. 



I 



COMPUTE FINAL CONFIDENCE-WEIGHTED 
DISPLACEMENT ESTIMATE. 



I 



CONVERT DISPLACEMENT ESTIMATE 
TO DEPTH ESTIMATE. 



FIG. 9 
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BACKGROUND OF THE INVENTION 
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For performing an optical measurement of depth, it is 
frequently desirable to derive a range image, which 
represents the distances between a viewing point and 
various surfaces in a scene. Range images may be de- 
rived by active techniques in which some form of en- 
ergy is directed toward an object and then measured on 
its return. Popular methods of active ranging include 
sonar^ laser rangefinding. and structured light. Range 
images may also be derived by passive techniques, in 
which the light from a normally illuminated scene is 
captured and analyzed by one or more television cam- 
eras. Of the passive techniques, binocular stereo is the 
most popular. 

There are many situations in which active techniques 
are inconvenient or ineffective in acquiring the desired 
range imagery. The most popular passive technique, 
binocular stereo, has a number of disadvantages as well. 
It requires the use of two cameras that are accurately 
positioned and calibrated. Analyzing the data involves 
solving the correspondence problem, which is the prob- 
lem of determining the matches between corresponding ?5 
image points in the two views obtained from the two 
cameras. The correspondence problem is known to be 
difficult and demanding from a computational stand- 
point, and existing techniques for solving it often lead to 
ambiguities of interpretation. The problems can be ame- 
liorated to some extent by the addition of a third camera 
(i.e. trinocular stereopsis), but many difficulties remain. 

The correspondence problem can also be avoided if 
one acquires a series of images from a series of closely 
spaced viewpoints, as is done in epipolar analysis. How- 
ever, the procedure in this case is quite cumbersome, 
since a camera must move along a trajectory over an 
extended period of time in order to gather a sequence of 
images. 



cal lenses arrayed in a rectangular or hexagonal lattice 
is preferred. 

A diffuser positioned between an object being imaged 
and the imaging elements is also provided in the pre- 
5 ferred embodiment. The diffuser acts as a low pass filter 
to reduce aliasing. A field lens is also used, and is posi- 
tioned between the converging lens and the imaging 
elements. The field lens redirects the light from the 
converging lens such that it appears to be focused from 
'0 infinity. A relay lens may also be used between the 
imaging elements and the photodetector array. The 
relay lens forms an image of the light imaged by. the 
imaging elements onto a screen. The relay lens relays 
the image from the screen to the photodetector array 
^5 which can be set back from the imaging elements. 
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SUMMARY OF THE INVENTION 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 iUustrates an optical ranging apparatus ac- 
cording to the present invention. 

FIG. 2 shows a CCD array and a preferred form of 
partitioning the array for the present invention. 

FIGS. 3A-3C illustrate the imaging of a point object 
by the present invention, when the object is in different 
regions of the object field. 

FIGS, 4A-4C demonstrate typical CCD sensor out- 
puts for the imaging tasks of FIGS. 3A-3C, respec- 
tively. 

FIG. 5A is an illustration of a first subimage formed 
by the present invention. 

FIG. 5B is an illustration of a second subimage 
formed by the present invention. 

FIG. 5C is an illustration of the subimage of FIG. 5A 
overlayed on the image of FIG^ 5B. 

FIG. 6A shows the geometry for a main lens focused 
on infinity as used with the present invention. 

FIG. 6B shows the geometry for a main lens focused 
at a distance D as used with the present invention. 

FIG. 7 shows a preferred embodiment of the ranging 
apparatus of the present invention. 

FIG. 8 an alternative embodiment of the ranging 
apparatus of FIG. 6 haying a relay lens. 

FIG. 9 is a flowchart showing the image processing 
steps of the present invention. 



DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Shown in FIG. 1 is an optical ranging apparatus 10 
having a main converging tens 12 receiving light from 



An optical depth resolver according to the present 
invention has a converging lens which receives light 
from objects in an object field. The lens directs the light 
to an image field of the converging lens where it is 45 
received by a plurality of imaging elements distributed 
about an image field of the converging lens. Each of the 
imaging elements receives light from the converging 
lens and fbrmis an image from the light it receives. 

The images from the imaging elements are received 50 an object 14 in an object field 16, and focusing the re- 

by a photodetector array which is divided into a plural- ceived light toward a lenticular array 18. The lenticular 

jtyofmacropixels,eachofwhichismadeupofaplural- array is an array of lenses (lenticules 20) aligned adja- 

ity of subpixels. Each macropixel of the photodetector cent one another in an image field 22 of the main lens 12. 

array receives the image from one of the imaging ele- Each lenticule 20 of the lenticular array 18 forms an 

ments, and each subpixel generates an electrical signal 55 imag;e from light received from the entire main lens 12 

indicative of the intenshy of light incident upon it. as seen from the location of that particular lenticule 20. 

Also provided with a preferred embodiment of the Since each lenticule 20 is in a different region of image 

present invention is a data processor which receives the field 22 of the main lens 12, each lenticule images light 

electrical signals from the subpixels and compares the from the entire surface of the main lens 12, but the light 

relative intensities of different subpixels to establish the 60 received by each lenticule is focused from a different 



depth of objects in the object field relative to a focal 
plane of the converging lens. In an alternative embodi- 
ment of the present invention, the imaging elements are 
in the form of an array of pinhole cameras, each of 
which images light from the entire surface of the con- 65 
verging lens. However, the preferred embodiment uses 
a lenticular array. The lenticular array may have cylin- 
drical lenses, but a two dimensional array using spheri- 



portion of the object field 16. The object field 16 of the 
main lens 12 is an area from which light is received by 
the main lens 12 and directed toward the lenticular 
array 18. Similarly, the image field 22 is the area to 
which tight received from the object field 16 is passed 
by the main lens 12. 

Positioned behind the leniiciilar array 18 is an array 
of charge coupled devices (CCD array 23) which func- 
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tion as photodetectors; FIG. 2 depicts a portion of CCD 
array 23 and demonstrates how the CCD array is ar- 
ranged for use in the present embodiment. The CCD 
array 23 consists of subpixels 24 each of which is a 
discrete photodetector and produces an output voltage 
proportional to the intensity of light incident upon it. 
The subpixels 24 are grouped in macropixels, each mac- 
ropixd containing a fmite number of subpixels 24. The 
macropixel regions of the CCD array are defined by the 
regions of the CCD array upon which the lenlicule 
images are incident. The pixels of the CCD array which 
are encompassed by one lenticule image make up all the 
subpixels of one macropixel. In FIG. 2, the double-line 
borders in the figure signify the separation between 
differcnl macropixels. Therefore, in the portion of array 
23 shown in FIG. 2, each macropixel has nine different 
subpixels 24. It will be understood that the double lines 
shown in FIG. 2 may not exist on the surface of the 
CCD array 23 itself, but serve to illustrate how groups 



intensity distributions of the macropixels receiving light 
has aii intensity peak in one of its subpixels. The loca- 
tions of the intensity peaks in the macropixels of the 
array 23 and the overall intensity distributions give 
5 information as to the location of the object 26 in the 
object field of the main lens. 

In FIG. 3C, the point object 26 is closer to the main 
lens 12 than the focal plane of the lens 12. This causes 
the focus of the lens 12 in the image plane to be beyond 
10 the lenticular array 18, and the light from object 26 does 
not come to a focus before reaching the array 18. Like 
the FIG. 3B example, this causes overflow of light into 
the lenticules adjacent the center lenticule. However, 
since the converging light has not yet come to a focus 
1 5 upon reaching the lenticular array 18, the distribution of 
the light from different subpixels is different than the . 
distribution resulting from the object being farther from 
the main lens 12 than the plane of focus (as in FIG. 3B). 
Specifically, for objects near the optic axis of the main 



of subpixels 24 are preferably arranged for processing. 20 lens 12, the intensity tends to be greatest at the subpixels 



In fact, the lenticular and CCD arrays may not be prop- 
erty aligned, but compensation for such misalignments 
may be provided by die data processing. 

Since each macropixel of the CCD array is positioned 
behind one of the lenticules 20 of the lenticular array 18, 25 
each of the macropixels receives an image which con- 
tains light passed by the entire main lens 12. However, 
the light received by each macropixel is received from 
a different angular position relative to the optic axis of 



closest to the central lenticule in FIG. 3C, and tends to 
be greatest at the subpixels away from the central lenti- 
cule in FIG. 3B. Further from the optic axis there is a 
shift in the distribution unless a Held lens is used as 
described below. 

Each of the subpixels across the dimension shown in 
FIGS. 3A-3C is labelled x, y, or z according to its 
location within a particular macropixel. Each of the 
subpixels x, y, and z corresponds to the light received 



the main lens 12. In the embodiment of FIG. 1, the 30 through a different portion of main lens 12. The particu- 



object 14 is shown as being perpendicular to the optic 
axis of the main lens 12, and in a plane of focus. Thus, 
any point along the surface of the object 14 is imaged by 
the main lens to be in focus at the lenticular array 18. 
Because of this focusing^ the light directed from any 
point on the object 14 is received by only one of the 
lenticules 20 of the lenticular array 18, and there is no 
overlap of light between lenticules. If, however, objects 
in the object field 16 of the main lens 12 are displaced 



lar macropixel in which the subpixels x, y, and z are 
located determines the region of the object field from 
whiph the light originates. Thus, the number of lenti- 
cules determines the spatial resolution of the received 
35 images. The number of subpixds per macropixel deter- 
mines the number of different "views" through the main 
lens 12, or the "motion resolution" or "parallax resolu- 
tion'* of the image at the CCD array. Each view 
through the lens is from a different angle, each subpixel 



relative to the focal plane of the lens 12, these objects do 40 group corresponding to one angle. 



not come to a focus at the lenticular array 18, and light 
from the same portion of the object field may be re- 
ceived by more than one adjoining lenticule 20. 
FIG. 3A shows main lens 12 having a point object 26 



To determine the position of an object in the object 
field of main lens 12, the subpixels of the CCD array are 
compared. FIGS. 4A-4C show the bar graphs from 
FIGS. 3A.3C, respectively. Underneath the bar graph, 



in a focal plane of the lens 12. As shown, the light from 45 in each of the figures, the intensity output for each of 



the object 26 comes to a focus at the lenticular array 18, 
and is therefore contained in only one lenticule 20. The 
bar graphs along the bottom of each of FIGS. 3A, 3B, 
and 3C show the relative intensities of light received by 



the subpixels x, y, and z is shown separately. If each of 
the subpixels of one type (e.g. x, y, or z) is assembled 
into a subimage, the result is three separate subimages, 
each representing a view through a different portion of 



each of three subpixels across one dimension of the 50 main lens 12. Thus, each subimage is a view of the entire 



55 



macropixels of the CCD array 23. This dimension is in 
the direction shown peipendicular to the optic axes of 
the lenses of the lenticular array 18. As shown in FIG. 
3A. the focusing of the light from the point object at the 
lenticular array 18 causes all the light received from the 
object to enter the center lenticule shown in the figure. 
Within the center lenticule, the light is evenly dispersed 
across all three horizontal subpixel locations of the 
center macropixel. 

The object 26 is shown again in FIG. 3B, but is be- 
yond the plane of focus in the object field. This causes 
the light from object 26 to come to a focus before reach- 
ing the lenticular array 18. Thus, the light diverges past 
the focal point and -*spills over^* into the lenticules 20 
adjacent the center lenticule. This "spillover" causes 65 
intensity distributions to be recorded in the macropixels 
adjacent the macropixel of the center lenticule. As can 
be seen from the bar graph of FIG. 3B, each of the 



object field as seen through a different *"viewmg aper- 
ture" of the main lens 12. By selecting all the x subpix- 
els, a virtual aperture through the right side of the lens 
is selected. By selecting all the z subpixels, a virtual 
aperture through the left side of the lens is selected. 

As shown in FIG. 4A, a point object in the focal 
plane of the main lens results in light being received in 
the subpixel groups x, y, and z in equal portions and 
only in the center macropixel. Thus, if one were to 
60 display the subimages of that point in sequence as an 
animation, the point would remain stationary. In FIG. 
4B (corresponding to FIG, 3B), it is shown that the 
intensity peaks output by subpixels of the subimages 
vary as the viewing aperture is changed from the x 
subimage to the y subimage, and vary further as the 
aperture is changed from the y subimage to the z subim- 
age. Peaks to the left of center (in the orientation shown 
in FIG. 4A) decrease as the viewing aperture is changed 
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from X to y to z (i.e. from left to right). However, the 
peaks to the right of center increase as the viewing 
aperture changes from the x subimage to the y subimage 
to the z subimage. Described in a different manner, the 
location of the highest pealc in the subimages shifts from 
left to right as the viewing apertures are changed from 
X to y to z. If viewed sequentially, a point somewhat oiit 
of focus (spreading across plural macropixels) would 
move from left to right with changing angle of view. 

In FIG. 4C (corresponding to FIG. 3C), left-of-ccn- 
ter peaks increase in intensity when the viewing aper- 
ture changes from left to right (x subiihage to z subim- 
age). Similarly, the right-of center peaks decrease in 
intensity when the viewing aperture changes from left 



through the right side of the main lens, while the image 
of FIG. SB is an image as seen through the left side of 
the main lens. The focal plane of the main lens is some- 
where near the center of the object 27, and one view, 

5 relative to the other, therefore appears as a rotation of 
the object 27 about an axis passing through the center of 
the object 27. FIG. 5C shows the image of FIG. 5A 
overlayed on the image of FIG. SB, demonstrating the 
relative displacement changes from one view to another 

10 of portions of the depicted object 27. These displace- 
ment changes are identified during processing of the 
images, and are used in establishing measurements of 
relative depth. 

. One example of a technique for determining the 



be described using FIG. 6A. Main lens 12 is focused on 
infinity, and has a focal length F. An object in the object 
field of the lens 12 is at a distance d from a plane of the 
lens. An image of the object forms at a distance g, 
where: 



to right: Thus, the location of the highest peak in the 15 depth of an object in the object field of main lens 12 can 

subimages shifts from right to left as the viewing aper- ' ' • " 

tures are changed from x to y to z. 

From the above it can be seen that the changing of 
peak intensities is opposite depending on whether the 
object 26 is behind or in front of the focal plane of the 20 
main Icris 12. Therefore, from detecting how these in- 
tensities change across the surface of the CCD array, an 
indication of the depth of the object in the object field 
can be obtained. 

Within a single macropixel receiving an image from a 25 
lenticule 20, each subpixel 24 detects light from a differ- 
ent portion of the main lens 12. The subimages, when 
assembled, therefore appear as views of objects in the 
object field from different angles. The parallax between 



By similar triangles, 



(I) 



(2) 



where v is the known aperture displacement determined 
by the number of subpixels in a macropixel. h is the 
each ofthe subimages gives information on the depth of 30 measured image displacement of the object, and e 



or, 



objects in the object field. When the subimages are 
viewed sequentially, the intensity differences appear as 
relative position changes of objects depicted in the imr 
ages. 

Because of the change in viewing apertures, the 35 
depth of other objects in the object field may be deter- 
mined by how much parallax exists relative to the focal g-Fj=>ih/v 
plane of the main lens 12. The more an object appears to rearranging, 
change position from one image to the next, the farther p^g^gh/v 
it is from the focal plane. Objects farther from the main 40 « 
lens 12 than the focal plane appear to change position 
toward the left when viewing the images in the order of 
left subpixel image through right subpixel image (x F=^(l-A/») 
subimage through z subimage). Conversely, objects 
closer to the main lens 12 than the focal plane appear to 45 therefore 
change position toward the right if the same viewing 
order is used. 

The multiple image example discussed above is illus- 
trative of the capabilities of the present invention to 
establish depths of objects in an object field. The num- 50 
ber of subpixels across one dimension of a macropixel 
gives the "parallax resolution" of the apparatus (how 
many difierent viewing angles are obtained). I^e num- 
ber of macropixels across one dimension of the array 
determines the "spatial resolution" of the apparatus in 55 
that direction (the number of different components into 
which the image plane is broken across that dimension). 
Although the system may be used to generate a se- 
quence of images for the purppse of presenting a scan of 
the object field, as suggested above, the data acquired 60 
from may also be used directly for the ranging of an 
object in the object field. 

FIGS. 5A and SB show two different views of an 
object 27 representative of two different subimages. 
The view of FIG. SA might be an "x" subimage re- 65 
ceived by the system of FIG. 1, while the view of FIG. 
SB might be a "z" subimage received by the system of 
FIG. 1. The subimage of FIG. SA is an image as seen 



equals the distance ig—F). Therefore, 



Vg^{UF%\~h/v) 



substituting into equation (1), 



(3) 



(4) 



(5) 



(6) 



(8) 



Therefore, by knowing F and v, and measuring h, the 
distance d can be calculated. 

In the case of a lens focused on a plane at a distance 
D, a distance calculation is made using the diagram of 
FIG. 6B. The lens of FIG. 6B has the following charac- 
teristics: 
F= focal length 
f= distance to lenticular array 
D= distance at which lens is focused 
d=: distance of object plane 
g= distance of image plane 
e^distance of image plane beyond lenticular array 
v= viewing position of aperture 
h=: displacement of object ray 
By similar triangles. 
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8 



<9) 



(10) 



(11) 



(12) 



10 



(13) 



15 



(14) 
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(15) 



(16) 



25 



CX:D array 23 of FIGS. 3A-3C to be skewed across the 
surface of the subpixels in each macropixel. Therefore 
the center of the lenticule image (and therefore the 
center of the macropixel) are displaced relative to the 
CCD array 23. Thus, subpixels on one side of a mac- 
ropixel may be overly illuminated due to the light being 
converging or diverging. The effects of such skewing 
are predictable and may be compensated for with pro- 
cessing techniques. However, the field lens removes the 
need for the additional processing by providing an opti- 
cal solution. 

The lenticular array 32 of FIG. 7 is a two dimensional 
lens array of spherical lenses, and therefore allows two 
dimensional motion analysis from the acquired subim- 
ages. In this embodiment, the lenses of the array are 
arranged in a rectangular lattice. However, a hexagonal 
or other type of lattice arrangement may be used as an 
alternative. The CCD array 33 is somewhat like the 
CCD array 23 of FIG. 1, and serves as an input to data 
processor 36, shown in block diagram form. Also in- 
cluded with the embodiment of FIG. 7 is a weak dif- 
fuser element 38 which diffuses the light from the object 
prior to its reaching the lenticular array 32. Without the 
diffuser, problems with aliasing might arise. Since the 
spatial resolution of the apparatus is determined by the 
number of lenticules, so is the sampling "frequency". If 
light from depth changes in the object field falls within 
an area which is smaller than the area of the object field 
from which light is received by a single lenticule, accu- 
rate resolution of the depth change can not be accom- 
plished. In other words, not enough samples exist to 
properly resolve the depth change and aliasing results 
(i.e. no useful information is provided about depth in 
that region). 

Diffuser 38, however, helps control aliasing by dis- 
persing the light received from the object field, thereby 
acting as a *'low pass filter". Some common examples of 
diffusers are glass elements with random surface varia- 
tions or diffracting optics. A weak, lenticular array may 
also be used. If a one dimensional lenticular array 32 
(cylindrical lenses) is used, a one dimensional diffuser 38 
may also be used, since only one dimension of depth 
changes need be resolved. In any event, the diffuser 
45 should ideally be in the aperture plane, and make a point 
spread function (PSF) of about 1 lenticule width. 
. As mentioned above, the lenticular array may be 
selected to be one dimensional (cylindrical) or two 
dimensional. A one dimensional array has better resolu- 
ular"amyTs^(irs7ribut'^"ac^^^^^^ 50 tion but only one direction of parallax A two dimen- 

particular macropixel. Thus, each macropixel is given a f^^^^' f ^^""^f. ^""^'1 *^Ph«"*^^' 

number of different virtual apertures of the lens in each l*^"ses instead of cyhndncal lenses. Such an array is 
of the two dimensions of the array. In the array shown preferably arranged m a rectangular or hexagonal lat- 
in FIG 2, all of the subpixels labelled xx are assembled t'^e. Such two dimensional screens are commercially 
into one subimage. all the subpixels labelled xy are ar- 55 available, or may be constructed by crossing two cylm- 



Since e=g— f, 
or 

rearranging, 
f^g-gfi/v 

or 

therefore, 

g«//(l-A/v) 



By the lens equation. 
Substituting eq. (15) into eq. (16), 
Using the relation 



(H) 



30 



(18) 



eq. (17) may be put in terms of D, such that it reduces to 
the following: 



35 



or 



I h/Fv-H/DK I - A/v) 



(19) 



40 



(20) 



Thus, knowing F, v, and D, and measuring h, the dis- 
tance d to a point on an object may be found. 

In a preferred embodiment of the invention, lenticu- 
lar array is used which has a two dimensional array of 
lenses to provide image data in each of two different 
dimensions. A CCD sensor such as that of FIG. 2 is 
used, and the image from one of the lenses of the lentic- 



ranged into a subimage, etc. In this manner, nine sepa- 
rate subimages are assembled, each of which consists of 
light having passed through a different region of main 
lens 12. 

A preferred embodiment, such as mentioned above, is 
shown graphically in FIG. 7. Main lens 12 is receiving 
light from an object 30 in the object field of the lens 12, 
and passing that light through to the image field of the 
lens 12. Located in the image field between the main 
lens 12 and lenticular array 32 is a field lens 34 which 
adjusts the focused light from the main lens such that it 
appears to have been focused from infinity. The absence 
of such a field lens 34 causes the light received by the 



60 



65 



drical screens. On any lenticular array, a finer array 
means better spatial resolution, but fewer viewpoints 
(subimages). 

One alternative embodiment of the present invention 
uses a relay lens to reimage the lenticular image onto a 
distant sensor array. Such an arrangement is shown in 
FIG. 8. Object 30 is shown being imaged by main lens 
12 through diffuser 38. Light from the main lens 12 is 
passed through field lens 34 and onto lenticular array 
32. However, instead of imaging immediately onto a 
CCD array, the image formed on the rear surface of the 
lenticular array is viewed by relay lens 40 which focuses 
the lenticular image onto photodetector array 42. An 
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optional piece of ground glass may be provided on the 
back of the lenticular array in on which a clear image 
may be fonncd. TTie combination of relay lens and pho- 
todctector array 42 might be in the form of a television 
camera assembly. This system allows more flexibility in 5 
how the images from the lenticular array are dispersed 
across the photodetector array 42; However, additional 
optics are required and the relative positioning of the 
elements make the system less stable. 

The present invention is unique in that it records a 10 
single monocular image, and uses that single image to 
calculate the depth of objects in the object field, Once 
the image is detected by the CCD array» the processing 
of the information is performed by a data processor. 
The data processor uses image processing techniques, 15 
many of which arc standard in the art of image process- 
ing. The following is a list of steps which may be fol- 
lowed in a preferred embodiment of the present inven- 
tion to accomplish the accurate processing of the ac- 
quired image information. 20 

Processing of the Image Data 

A n^imber of different processing methods may be 
used to extract depth information from the recorded 
image data. The preferred embodiment uses a sequence 25 
of genera! steps which are shown in the blocks of the 
flowchart of FIG. 9. First the image received by the 
CCD array is digitized (block 50). The intensity of the 
light received by each pixel of the CCD array results in 
the generation of an analog voltage which is converted 30 
to a digital code by an analog-to-digital (A/D) con- 
verter. The number of quantization levels available with 
the A/D converter depends on how many bits are used 
to represent each pixel. This in turn determines the 
number of distinct grey levels which are recognized by 35 
the data processor. 

After digitization, the subimages corresponding to 
different viewpoints are extracted (block 52). For this 
step, the image is multiplied by a weighting mask to 
select individual subimages. Preferably, the lenticular 40 
array is aligned with the CCD array such that the image 
from each lenticule is evenly distributed across one 
macropixel, therefore encompassing an even number of 
subptxels. In such a case, a mask is defined which, for 
each macropixel, multiplies all the subpixels by zero 45 
except for the subpixel corresponding to the desired 
subimage. For each different subimage, a different sub- 
pixel for each macropixel is selected. Thus, for each 
position, the mask blocks out all but one set of subpixels, 
such that each subimage consists of one subpixel from 50 
each macropixel. Each subimage therefore represents a 
different virtual aperture position (viewing light from a 
different portion of the lens). 

In some cases, the lenticular array may not be exactly 
aligned with the CCD array, and a weighting and aver- 55 
aging technique is used to generate the desired subim- 
ages. In this case, it is not possible to generate the subim- 
age by selecting individual pixel values, because the 
effective position of the desired pixel may lie at a frac- 
tional position between CCD elements. Therefore, it is 60 
necessary to estimate the value of the virtual pixel at a 
fractional position. This can be achieved by taking a 
weighted average of the observed pixels surrounding 
the virtual pixel and using well known interpolation 
techniques such as bilinear interpolation. 65 

Once the desired subimages are assembled, each 
subimage is enhanced through well known image pro- . 
cessing techniques (block 54). In the preferred embodi- 



ment, this enhancing includes preprocessing the subim- 
ages to remove low-spatial-frequency variation that 
may result due to non-uniformity in the optical system 
or sensor sensitivity. This may be accomplished with a 
broadband high-pass filter which, for each pixel of a 
subimage, subtracts a local average of image intensity 
taken from sunotinding pixels. Alternatively, a local 
automatic gain control process may be used which di- 
vides regions of the subimage by a local average. 

Displacement analysis is then performed (block 56) to 
measure object displacement between image pairs. This 
analysis uses techniques typically used for motion anal- 
ysis or stereo analysis. One possible technique is to 
perform a least-squares match as commonly understood 
in the art of image processing. Such a least-squares 
matching technique is demonstrated by Lucas and 
Kanade in An Iterative Image Registration Technique 
with an Application to Stereo Vision, Proceedings of 
Image Understanding Workshop, April 1981, pp. 
121-130. Using this technique, the displacement esti- 
mate is given by: 

where Ix is the spatial derivative of the image intensity 
in the x direction and I^is the derivative of intensity as 
the viewing position is moved in the x direction of the 
aperture. Summation is taken over a small spatial patch 
within the image. Larger patches lead to estimates with 
reduced noise but tend to smooth out sharp changes in 
displacement. The output using this technique is an 
image with a displacement estimate at each position. 
Following the above least squares method, a confidence 
factor of 2[(Ix)^] may also be assigned to each local 
estimate. 

In processing the image data in two dimensions, the 
previously mentioned least squares technique, described 
for subimages in one dimension, can similarly be used 
for subimage data in a second dimension. Once this is 
complete, the final confidence-weighted estimates are 
computed (block 58). Displacement estimates are ex- 
tracted for all adjacent image pairs. On the assumption 
that the magnitude of the displacements in one dimen- 
sion should equal the magnitude of the displacements in 
the second dimension, all of the displacement estimates 
in both dimensions are combined into a single displace- 
ment estimate. This is accomplished by weighting each 
displacement image by its corresponding confidence 
image. 

Once a final displacement estimate is acquired, the 
displacement estimate is converted to a depth estimate 
(block 60) using the geometry shown in FIGS. 6A and 
6B. This depth estimate output is then applied to the 
particular application for which the invention is being 
used. 

One processing consideration which arises in practic- 
ing the present invention is the alignment of the desired 
macropixels with the actual subpixels of the photode- 
tector array. In the aforementioned case of the lenticu- 
lar array not lining up evenly with the subpixels of the 
CCD array, it is necessary to generate a weighting mask 
to be used during processing. The following is a pre- 
ferred method for generating such a mask. 

A uniformly illuminated white card is first placed in 
front of the main lens in the object field. The aperture of 
the main lens is then reduced so that the macropixels 
include. only a few sensor subpixels. The image is digi- 
tized, and the locations of the peaks in the macropixels 
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are determined. A weighting mask is then defined in 5. An optical depth resolver according to claim 4 

which the smooth weighting functions underneath each wherein the spherical lenses are arrayed in a rectangular 

macropixel are centered on the correct center of the lattice. 

macropixel. 6. An optical depth resolver according to claim 1 

While the invention has been particularly shown and ^ wherein each of the imaging elements receives light 

described with reference to a preferred embodiment from the entire surface of the converging lens, 

thereof, it will be understood by those skilled in the art 7. An optical depth resolver according to claim 1 

that various changes in form and details may be made wherein the photodetector array is a CCD array, 

therein without departing from the spirit and scope of 8. An optica] depth resolve according to claim 1 

the invention as defined by the appended claims. In wherein each subimage is representative of light passing 

particular, the specific processing techniques may vary through a different region of the converging lens, 

depending on the application of the invention. The least 9* An optical depth resolver according to claim 8 

squares method discussed above is but one image pro- wherein the data processor assembles the subimages and 

cessing technique understood by those skilled in the art. compares the parallax from one subimage to another in 

Furthermore, the lenticular array may be replaced by determining depth of objects in the object field, 

another form of imaging device such as a pinhole array. 10. An optical depth resolver according to claim 8 

The photodetector array may also be a type of photode- wherein the shift of intensity peaks from one subimage 

lector other than a CCD device. to another is detected by the data processor and used 

The data which is gathered by the optical system may determining depth of objects in the object field, 

be processed for various purposes. For example, the An optical depth resolver according to claim 1 

data can be applied to a three dimensional display. The wherein there is one macropixel for each image gener- 

subimages may be viewed individually or sequentially ated by an imaging element. 

to provide two dimensional projections from different An optical depth resolver according to claim 1 
angles. The range data obtained from processing the 25 ^'^^e'"^*" ^^^a processor runs a least squares process- 

subimages may be used to identify objects in the object ^8 method in making determinations of depth of ob- 

field. A system using such a ranging technique may also J^^^s the object field. 

be used for such applications as setting off security An optical depth resolver according to claim 1 

alarms or for automatic navigation (obstacle avoid- further comprising a weak diffuser which diffuses the 
ance). ^ 30 ^^8^t from objects in the object field prior to the light 

I claim: ^ng received by the imaging elements. 

1. An optical depth resolver, comprising: An optical depth resolver according to claim 13 

a converging lens receiving light from objects in an wherein the diffuser is a piece of glass with random 

object field and directing the received light to an surface variations. 

image field of the converging lens; 35 optical depth resolver according to claim 1 

a plurality of imaging elements distributed about the further comprising a field lens positioned between the 

image field of the converging lens in the image converging lens and the imaging elements, and red irect- 

field of the lens such that each imaging element the hght from the converging lens such that the 

receives Ught from the converging lens and forms Jl^^^ ^^^^ converging lens appears to have been 
an image from the light it receives; 40 focused from mfimty. 

a photodetector array divided into a plurality of mac- ^ ^^^^^^ ^®P^^ resolver according to claim 1 

ropixels each of which is made up of a plurality of compnsmg a relay lens between the imaging 

subpixels. each macropixel receiving an image e^*'?^^"*? and the photodetector array, the relay lens 

formed by one of the imaging elements such that ^d^^cting the images from the imaging elements to the 
each subpixel of a macropixel receives light from « Photodetector array. 

one portion of the lens as imaged by one of the ^^^P^^^f ^fP^^^ ^"^l^er aa;ordmg to claim 16 

imaging elements, each subpixel generating an elec ^^f.^^" ^^^^y ^^"^ '^^ocuscs the images generated 

trical signal indicative of the intensity of light ind. Vh^'IT^^S^TT' i ^- . , • 

dent upon if and ^ * 18, An optical depth resolver according to claim 17 

.data processor receiving the electrical signals from » dSAtS^.trf.^ 

tbesubpixelsandprocessingthesignalsf^™ t^l d^^tS^^SS^^^^ 

ent subpuels. he dau processor .dentifymg subim- optical depth r^olver a^rftaj o claim 16 

^^i„ H S^""! ^i"" T'' therein the%elay lens is part of a television cTnTe^a. 

^r^^« .1.1 I^ir. « Tn'^T'r, P«'r°T"^' « 20. An optical depth resolver as claimed in claim 1 

across the mwge field, a plurality oflocal displace- therein the data processor estimates arc based on de- 

m«.t esnmates of image displacement between rfvatives of local intensity between subimages for 

subimages. to create a range image of depths of patches of macropixels 

<*jects across the object field. 21. An optical depth resolver as claimed in claim 20 

2. An optical depth resolver according to claim 1 « wherein the estimates are based on derivatives in two 
wherem the plurality of imaging elements comprises a dimensions. 

lenticular array. 22. An optical depth resolver as claimed in claim 1 

3. An optical depth resolver according to claim 2 wherein the data processor applies a weighting mask to 
wherem the lenticular array comprises an array of cy- the electrical signals generated by the photodetector 
lindncal lenses. 65 array in order to adjust the signals for misalignment of 

4. An optical depth resolver according to claim 2 the imaging elements with the photodetector array, 
wherein the lenticular array comprises an array of 23. An optical depth resolver as claimed in claim 1 
spherical lenses. wherein the data processor processes subpixels of at 
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least three subimages in performing each local displace- image field of the converging lens, each imaging 

ment estimate. element forming an image from light it receives; 

24. An optical depth resolver comprising: receiving the images formed by the imaging elements 
a converging lens receiving light from objects in an with a photodetector array divided into a plurality 

object field and directing the received light to an 5 of macropixels each of which is made up of a plu- 

image field of the converging lens; rality of subpixels, each macropixel receiving an 

a lenticular array having a two dimensional array of image formed by one of the imaging elements such 

lenticules distributed about the image field of the that each subpixel of a macropixel receives light 

converging lens such that each lenticule receives from one portion of the lens as imaged by one of 

light directed from a different portion of the object 10 the imaging elements, each subpixel generating an 

field and receives the directed light at a different electrical signal indicative of the intensity of light 

angle relative to an optic axis of the main lens, each incident upon it; and 

lenticule forming an iniage from the light it re- receiving the electrical signals from the subpixels 

ceives; with a data processor and processing the signals 

a photodetector array divided into a plurality of mac- 13 from different subpixels to create a range image of 

ropixels each of which is made up of a plurality of estimates of depths of objects across the object 

subpixels, each macropixel receiving the image field. 

formed by one of the lenticules such that each 33, A method according to claim 32 wherein receiv- 

subpixel of a macropixel receives light from one ing said light from the converging lens with a plurality 

portion of the converging lens as imaged by one 20 of imaging elements comprises receiving said light from 

lenticule, each subpixel generating an^ electrical the converging lens with a lenticular array, 

signal indicative of the intensity of light incident 34. A method according to claim 32 wherein receiv- 

upon it; ing said light from the converging lens with a plurality 

a data processor receiving the electrical signals from of imaging elements comprises receiving said light with 

the subpixels and processing the subpixel signals to 25 the imaging elements such that each imag;ing element 

detect variations in intensity of light directed from receives light from the entire surface of the converging 

a plurality of locations in the object field relative to lens. 

the portion of the converging lens through which 35. A method according to claim 32 wherein receiv- 

the light passes, the data processor identifying ing the images formed by the imaging elements further 

subimages of subpixels having like spatial positions 30 comprises arranging the subpixels of each macropixel of 

within different macropixels and performing, the photodetector array in a similar positional distribu- 

across the image field, estimates of depths of ob- tion such that a group of subpixels, each from a similar 

jects across the object field based on local deriva- position in a different macropixel, together form a 

tives of intensity between subimages to create a subimage representative of light passing through one 

range image. 35 region of the converging lens. 

25. An optical depth resolver according to claim 24 36. A method according to claim 35 wherein process* 
wherein the photodetector array is a CCD array. ing the electrical signals from different subpixels further 

26. An optical depth resolver according to claim 24 comprises assembling said subimages and comparing 
further comprising a weak diffuser diffusing light be- the parallax from one subimage to another in determin- 
tween the object field and the lenticular array. 40 ing depth of objects in the object field. 

27. An optical depth resolver according to claim 24 37. A method according to claim 35 wherein process- 
further comprising a field lens between the converging ing the electrical signals from different subpixels funher 
lens and the lenticular array, the field lens redirecting comprises detecting the shift of intensity peaks from one 
the light from the converging lens such that it appears subimage to another in determining depth of objects in 
to have been focused from infinity. 45 the object field. 

28. An optical depth resolver according to claim 24 38. A method according to claim 32 further compris- 
further comprising a relay lens between the lenticular ing diffusing said light from objects in the object field 
array and the photodetector array, the relay lens redi- with a weak diffuser. 

reeling the images from the lenticular array to the pho- 39. A method according to claim 32 further compris* 

todetector array. 50 ing redirecting the light from the converging lens to the 

29. An optical depth resolver as claimed in claim 24 imaging elements with a field lens. 

wherein the estimates are based on derivatives in two 40. A method according to claim 32 further compris- 

dimensions! ing redirecting the images from the imaging elements to 

30. An optical depth resolver as claimed in claim 24 the photodetector array with a relay lens. 

wherein the data processor applies a weighting mask to 55 41. A method as claimed in claim 32 wherein the data 

the electrical signals degenerated by the photodetector processor estimates are based on derivatives of local 

array in order to adjust the signals for misalignment of intensity between subimages for patches of macropixels. 

the lenticular array with the photodetector array. 42. A method as claimed in claim 41 wherein the 

31. An optical depth resolver as claimed in claim 24 estimates are based on derivatives in two dimensions, 
wherein the data processor processes at least three 60 43. A method as claimed in claim 32 wherein the data 
subimages to perform each estimate. processor applies a weighting mask to the electrical 

32. A method of making depth measurements, com- signals generated by the photodetector array in order to 
prising: adjust the signals for misalignment of the lenticular 

providing a converging lens receiving light from array with the photodetector array, 

objects in an object field and directing the received 65 44. A method as claimed in claim 32 wherein the data 

light to an image field of the converging lens; processor processes at least three subimiages to deter- 

receiving said light from the converging lens with a mine each depth of an object, 

plurality of imaging elements distributed about the » » ♦ • • 
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